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Abstract 

We interviewed eight University of Toronto (U of T) instructors who have offered MOOCs on 
Coursera or EdX between 2012 and 2014 to understand their motivation for MOOC instruction, 
their experience developing and teaching MOOCs, and their perceptions of the implications of 
MOOC instruction on their teaching and research practices. Through inductive analysis, we 
gleaned common motivations for MOOC development, including expanding public access to high 
quality learning resources, showcasing U of T teaching practices, and attempting to engage 
MOOC learners in application of concepts learned, even in the face of constraints that may 
inhibit active learning in MOOC contexts. MOOC design and delivery was a team effort with 
ample emphasis on planning and clarity. Instructors valued U of T instructional support in 
promoting systematic MOOC design and facilitating technical issues related to MOOC platforms. 
The evolution of MOOC support at U of T grew from a focus on addressing technical issues, to 
instructional design of MOOCs driven, first, by desired learning outcomes. Findings include 
changes in teaching practices of the MOOC instructors as they revised pedagogical practices in 
their credit courses by increasing opportunities for active learning and using MOOC resources to 
subsequently flip their classrooms. This study addresses the paucity of research on faculty 
experiences with developing MOOCs, which can subsequently inform the design of new forms of 
MOOC-like initiatives to increase public access to high quality learning resources, including those 
available through U of T. 
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Introduction 

We studied the experiences of University of Toronto (U of T) instructors who have offered 
Massive Open Online Courses (MOOCs) on Coursera or edX platforms between 2012 and 2014. U 
of T initiated institutional partnerships with Coursera in 2012, and EdX in 2013, to deliver a 
number of MOOCs. This MOOC activity occurred under the umbrella of Open UToronto, a 
broader institutional initiative established in the spring of 2012 to promote discovery, use, 
creation, and sharing of open access literature, shared content and educational resources, as well 
as open courses. 

Expanding previous U of T MOOC research (Chen, Haklev, Harrison, Najafi, & Rolheiser, 2015; 
Restoule, 2013), this timely study addresses the paucity of research on faculty members’ 
experiences with creating MOOCs (Liyanagunawardena, Adams, & Williams, 2013) beyond 
descriptive accounts of MOOC design (Robinson, & Ash, 2014; Severance, 2013). 


Background 

Empirical literature on faculty members’ MOOC development experience is sparse. Consequently, 
we drew on online learning literature to understand issues that faculty members recount 
regarding 1) motivations to teach in a different context (face-to-face vs. online), 2) instructional 
redesign, and 3) impact on their teaching practices. 

Motivational Factors 

Personal interest, undertaking a professional challenge, and the potential for increasing students’ 
access to education were among intrinsic motivators for faculty members to teach online courses 
in Maguire's (2005) review of 13 studies. Haklev (2011) interviewed five faculty members in two 
Chinese universities that participated in a project to improve undergraduate level education 
through engaging faculty members in designing open educational resources (OERs). Faculty 
members’ motivation to apply for the program included institutional recognition and using the 
application process to improve course design and student learning. 

In the MOOC context, Holland and Tirthali (2014) interviewed 83 individuals, including 
administrators and faculty members from institutions active in MOOC initiatives, to explore 
institutional goals for engaging in such initiatives. Increasing global access to education through 
an outreach beyond physical and geographical boundaries and enabling broad access to unique 
faculty expertise were among stated goals. 
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Instructional Design Considerations 

Numerous considerations guide the instructional design of MOOCs. Instructional design 
frameworks motivated by desired learning outcomes—e.g. Understanding by Design (Wiggins, & 
McTighe, 2005) and Integrated Course Design (Fink, 2013)—facilitate online and face-to-face 
course design. Adaptations of these frameworks, for example, have informed MOOC design at U 
of T (Open UToronto MOOC Initiative: Report on Second Year of Activity, 2014). As another 
example, MOOC Canvas (Alario-Hoyos, Perez-Sanagustin, Cormier, & Kloos, 2014), provides a 
visual participatory framework to guide MOOC design in two steps: assessing available resources; 
and making design decisions based on available resources. 

A team-based approach increasingly characterizes online course design where instructional 
designers, technical support staff, and, sometimes, graduate students, collaborate with faculty 
members (Koehler, Mishra, Hershey, & Peruski, 2004). Such collaboration may spark peer review 
and reflection opportunities absent when instructors work individually (Haklev, 2011). MOOCs 
are mostly developed by teams (Alario-Hoyos et ah, 2013; Holland, & Trithali, 2014) or taught by 
teams (Arnold, Kumar, Thilosen, & Ebner, 2014). 

Time is another important consideration in MOOC instructional design. Teaching online credit 
courses and MOOCs require both advance planning and continuous presence to provide feedback 
to students during the course (Arnold et ah, 2014; Power, 2009). The time commitment and 
resource allocation for MOOCs are reportedly higher than for credit courses, including time when 
faculty members create MOOCs, and later, when they re-design their credit courses to leverage 
MOOC resources (Holland, & Trithali, 2014). 

Impact on Practice 

The 913 faculty members who participated in Shea, Picket, and Li’s (2005) study answered a 
question about the impact of online course design and teaching on their pedagogical assumptions 
and classroom teaching. A regression analysis suggested opportunities for learning, including 
alternate means of instruction and assessment, as factors significantly related to participants' 
satisfaction with online instruction. 

In another three-round action-research study following seven faculty members who designed 
online courses, findings suggested that opportunities for reflection encouraged them to consider 
alternate views of teaching and learning (McQuiggan, 2012). Haklev (2011) reported that faculty 
members who developed open educational resources as part of their involvement in the Top Level 
Quality Project redesigned their lectures to include discussion, provided on-demand access to 
lectures, or leveraged social media to communicate with students. 

Similarly, MOOCs could encourage revising pedagogical practices and/or credit course redesign. 
Course redesign may result in a flipped classroom format using MOOC material or integrating 
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frequent feedback, discussion, and peer-assessment within the curriculum (Holland & Tirthali, 
2014). 

While faculty members' experiences with designing and teaching online credit courses are well 
documented, similar studies regarding MOOCs are missing. Accordingly, three research questions 
guided our study, corresponding to the dimensions of faculty engagement noted previously in this 
paper: 1) motivations, 2) instructional redesign, and 3) impact on practice. 

• Research question one: What motivated U of T instructors to offer MOOCs? 

• Research question two: How did U of T instructors design, develop, and deliver their 
MOOCs? 

• Research question three: What were the implications of MOOC instruction for instructors 
as researchers and instructors? 


Study Design and Background 


Participants 

As of September 2014, U of T instructors taught 12 distinct MOOCs, some re-offered, on Coursera 
and edX (Open UToronto MOOC Initiative: Report on Second Year of Activity, 2014). A call for 
study participation was sent to 12 U of T MOOC instructors, and eight agreed to participate. In 
compliance with our U of T ethical approval, the identity of the participants is kept confidential 
and no reference is made to their name, MOOC title, or exact time period of MOOC offering. We 
refer to research participants as "instructors". 

MOOC Instructional Design Support at U of T 

The following contextual information is relevant to the study focus and how U of T instructors 
were supported in their MOOC design process. Since 2012, the Online Learning Strategies (OLS) 
portfolio at U of T has been providing instructional support for MOOC development. OLS 
supports MOOC instructors in the following ways: 

• advising on evolving institutional strategies and new developments; 

• delineating institutional MOOC development workflow; 

• outlining critical success factors and consulting on resourcing implications; 


This work is licensed under a Creative Commons Attribution 4.0 International License. 


236 



University of Toronto Instructors' Experiences with Developing MOOCs 
Najafi, Rolheiser, Harrison, and Haklev 


• conducting team-based MOOC design workshops to systematically facilitate an 
instructional design process that targets: (a) setting clear weekly learning outcomes; (b) 
designing assessments that align with learning outcomes; and (c) developing learning 
activities and resources that allow learners to develop knowledge and competency as 
relevant to learning outcomes; and 

• organizing symposia and round-table sessions to disseminate findings from faculty 
MOOC research and raise awareness about the latest developments in U of T MOOCs. 

OLS also provides on-demand technical and instructional coaching and collaborates with MOOC 
platform providers to prepare necessary technical support for MOOC instructional teams and 
solve technical issues as they may arise. 

Methods 

Semi-structured interviews with participants were conducted between August 2013 and August 
2014. Other data sources included OLS documentation and curricular material from U of T 
MOOCs available on Coursera or edX. An inductive approach was used to analyze interview data 
(Creswell, 2012; Thomas, 2006) by: 

• coding interviews and transcribing code segments relevant to research questions; 

• collapsing codes into emergent themes and categories; 

• corroborating interview data with other data sources; and 

• preparing descriptive accounts of major and minor themes from the data. 


Results 

Findings of the study are presented here, under six main themes. Table 1 illustrates how the six 
main themes of findings map onto our three research questions. In the sections that follow, we 
present each main theme and provide interview excerpts to further contextualize the findings. 
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Table 1 

Findings: Mapping of the Six Themes to Three Research Questions 


Findings: Main Themes 


Instructors' 

MOOC 

MOOC 

Measures MOOC 

Implications 


motivations 
to offer 
MOOCs 

design 

development 
and offering 

for MOOC development 
success support 

of MOOC 
instruction 

Research 

question 

/ 





one 






Research 

question 

two 


/ 

/ 

/ / 


Research 





/ 

question 

three 







Motivation to Offer MOOCs 

Contributing to open educational resources (OERs). 

All U of T MOOCs were offered free of charge in 2012 and 2013. U of T’s positioning of MOOCs as 
being accessible to a broad audience as open courses was the main motivation for instructors to 
offer MOOCs. Through MOOCs, instructors contributed to publicly available OERs and, as one 
instructor mentioned, also showcased the quality of teaching and learning at U of T: 

When I heard about MOOCs I also thought that U of T is a big 
mysterious place for people who walk by and they may think 
some holy mysterious learning is happening here. I think there is 
wonderful learning happening at U of T. Always thought we 
could do more to open U of T to the public. [Developing a] 

MOOC was a tangible way to do that. 

Upon conclusion, all U of T MOOCs are left archived on their respective platform. Designing on- 
demand MOOCs was another step forward in providing public access to such OERs, an instructor 
suggested. The structured course-like format of MOOCs was cited by one instructor as enabling 
learning about a topic by covering interconnected concepts in a progressive manner. 
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Academic relevance. 

According to instructors, with global learners signing up for MOOCs in massive numbers, their 
teaching and its impact would transcend the limits of U of T credit courses. Beyond being a 
professional desire for some instructors, such international exposure allowed the instructors to 
examine the academic and instructional potential of MOOCs, and in a broader sense, online 
learning at large scale. 

Experiencing firsthand the complications of teaching in a MOOC context and assessing learners 
within the constraints of MOOC platforms were frequently cited motivations for offering MOOCs. 
Two instructors indicated the anticipated pedagogical and academic implications of MOOCs as 
their motive. From a pedagogical perspective, understanding how to foster deep learning in 
MOOCs informed the instructional design of large credit courses, as suggested by one instructor: 

In small classes students get to practice more skills. As the class 
size grew, the number of skills practiced in class decreased and a 
lot of courses have become lecture and multiple-choice 
questions. So I thought MOOCs were interesting because if you 
could show that at that scale you could bring in the skills, then 
people who teach 200-student classes will not have an excuse. It 
could be a driver for a change. 

MOOCs could also facilitate scaling up of knowledge within new fields of study by sharing current 
knowledge and enabling idea exchange that leads to inter-institutional research partnerships. 

MOOC Design 

Instructional team. 

MOOC design, development, and delivery was a team effort where instructors collaborated with 
teaching assistants, instructional designers, and technical staff to design and implement MOOCs 
on a platform, and monitor learner activities while the MOOC was running. In most cases, the 
instructors developed the bulk of the MOOC content. 

Learning outcomes. 

MOOC design was mainly driven by pedagogical approaches deemed effective in credit courses in 
their respective discipline. The overarching design goals for U of T MOOCs ranged from providing 
individual learners with as many opportunities as possible to practice what they learned, to 
deliberately fostering a sense of community, and to maximizing engagement in discussions. 
Regardless of discipline, instructors stressed the importance of learning relevance to MOOC 
learners and envisioned opportunities for self-reflection. 
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Instructors stated clear learning goals for their MOOCs. We synthesized high-level desired 
learning outcomes for U of T MOOC learners into three themes: 

• understand the fundamental concepts of a discipline; be able to transfer knowledge to 
other contexts; and apply concepts to relevant situations to improve existing practice; 

• become familiar with methods of scientific inquiry, problem solving, and reasoning 
relevant to a discipline; be able to apply those methods to their own inquiry; develop 
competency to assess the quality of existing work in their relevant context; and 

• develop knowledge of the discourse and vocabulary of a discipline and be able to use both 
in their own learning/work/personal contexts. 

The instructors also described desired learning outcomes for students taking their credit course as 
the ability to: 

• apply knowledge to new situations; being able to transfer knowledge; 

• articulate fundamental concepts of the discipline and explain them to their peers; 

• develop critical thinking and inquiry skills and be able to locate relevant resources to 
enrich their learning without being highly dependent on an instructor; and 

• develop metacognitive and self-regulative skills. 

Instructors' desired learning outcomes for their MOOCs and credit courses showed considerable 
overlap. 


MOOC learners. 

When asked about their target audience, instructors expressed initial uncertainty about actual 
MOOC learners and their academic or professional backgrounds. Instructors designed the 
MOOCs for a perceived group of learners who would find the MOOCs learning outcomes relevant. 
The target audience for U of T MOOCs ranged from learners with little or no previous knowledge 
or experience, to professionals who desired to expand their knowledge on a topic. One instructor 
indicated how the diversity of students in a credit course, with regards to knowledge of course 
subject, guided the perception of MOOC learners: 

I wanted to plan a course accordingly for people who had little 
knowledge to people who could know more than me and could 
teach the course. It is the balance of start where you are in terms 
of knowing, and push yourself to go deeper. Make this an 
opportunity to deepen your knowledge about something. 
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Generally, learners who registered in U of T MOOCs were internationally distributed. OLS 
statistics, as of July 2013, regarding the geographic distribution of U of T MOOC registrants 
confirmed instructors’ perception of learner diversity: Australia (1,268); Africa (1,610); South 
America (3,244); Asia (9,938); Europe (15,655); and North America (16,894). 

MOOCs targeted towards learners with no or little knowledge in the field also attracted highly 
educated learners. Reflecting on the reasons why learners with postgraduate degrees would enroll 
in a MOOC that covered fundamental concepts of a discipline, an instructor stated: 

From the discussion forum, I realized that a lot of people, even if 
they had taken [courses relevant to this subject] before, never 
really learned those learning goals or did not learn them very 
well, and many of them found it productive to go back and revisit 
them. Perhaps, they have taken more advanced courses or need 
it for a job. 

MOOC Development and Implementation 

Some instructors had online teaching experience. However, we could not find any evidence that 
instructors with no online teaching experience were at a disadvantage while planning and offering 
MOOCs. 

Instructors took different approaches to plan and develop MOOCs. Owing to the short time 
available between the approval of some of the MOOCs and their launch date, some instructors 
produced the latter parts of their MOOCs while the course was running. The instructor of an 
earlier U of T MOOC explained: 

We agreed to offer the MOOC [within three months from the 
agreement]. Timeline was incredibly tight. We were developing 
the MOOC as it was being offered. We started two weeks ahead 
so the first two weeks were done and then we were just go, go, 
go! 

Alternatively, instructors completely developed their MOOCs before the launch date when the 
approval date and the launch date were further apart. In both cases, according to the instructors, 
the process of MOOC development, including outlining learning outcomes, content creation, and 
assessment design, was time consuming and labor intensive. 

Two planning and production strategies deemed as effective by the instructors were: (a) a 
production line approach, and (b) an intensive plan-ahead approach. A production line approach 
was described as: "I first created lectures, then somebody edited those, somebody else generated 
questions on the video, and somebody else uploaded the video lecture with the embedded 
question. This system was effective. Each person was ahead of the next person." In an intensive 
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plan-ahead approach, all components of the MOOC were completely planned and scripted before 
the production started. 

To facilitate the structural examination of a MOOC, we informally divide it into three 
components: learning, assessment, and communicative. 

MOOC learning components: Keeping content concise. 

Learning components included video lectures, readings, guest speakers, and external links. U of T 
MOOCs were self-contained with no required textbook. Depending on disciplinary practice, some 
instructors integrated, or intended to integrate, additional technological tools to enrich learners’ 
experience. Such integration demanded close collaboration between instructors and MOOC 
platform providers and was not always feasible due to time constraints or technical complications. 
Instances of successful integration existed where, for example, MOOC learners engaged in hands- 
on hypotheses testing and theorizing using community-generated data. 

Short video lectures. According to the instructors, video lectures were a central 
learning component in U of T MOOCs. A common theme regarding MOOC video lectures was 
their much shorter length compared to credit course lectures. Although a challenging medium to 
convey concepts thoroughly, instructors felt that video lectures suited perceived learning 
outcomes of MOOC learners. As one instructor stated: “[MOOC students] have busy lives. For 
them, 15-minute bites can work in a day. So for purposes of people being able to keep up with a 
MOOC, this would work.” Instructors planned their video lectures to be precise and engaging, 
with opportunities for learners to reflect on content through in-video quizzes. This work required 
advance planning and, in many cases, scripting. 

One instructor, however, complemented instructor-dominated video lectures with interviews 
involving other subject-matter experts: 

Others had said that they didn’t like the talking head... I also 
thought of the possibility with the MOOC is that so many people 
can be part of the teaching and updating of material. So, one of 
the first thoughts I had was we can include other professors that 
do [name of the field] research and I invited them all to do a 
lecture. It turned out we did interviews instead to make it more 
comfortable... ask them the questions that I am hoping the 
students would ask... 

The variety in video production was largely affected by discipline and by the desired learning 
outcomes of a MOOC. Interviews with experts, as mentioned here, were not relevant to all 
MOOCs. 
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Instructors approached video production in different ways. Some instructors recorded their 
videos in a DIY format, others had technicians to produce professional videos, while others had 
partial technical assistance. Video production approaches depended on various factors, including 
personal preference, technical knowledge, time constraints, departmental facilities, and to some 
extent, available funding. 

MOOC assessment components: Scaffolding complexity. 

Assessment components allowed learners to apply their learning in practice, receive feedback in a 
variety of formats, and collect grades that would count towards obtaining a certificate of 
completion, if desired. Assessment design closely followed the learning outcomes of a MOOC and 
emphasized application of knowledge and reflection over information recall. While 
acknowledging the apparent limitations of assessing learning in MOOCs, the instructors utilized 
strategies to leverage available assessment tools relevant to their learning goals. One instructor 
stated: “Quite happy with the set of questions we made. Managed to find other ways of asking 
questions within the constraints and getting students to think and getting them to apply 
knowledge.” Another instructor explained why social sciences MOOCs may face further challenges 
regarding assessment design: "Social science courses required a different approach for 
assessment. For [more technical] MOOCs you could design tests that were completely computer 
graded. There were peer-assessment pieces but usually it was a very digital thing." In other words, 
some disciplines could utilize assessment approaches that had time efficiencies related to the 
technology tools utilized. 

For information recall purposes, instructors used multiple-choice questions and gauged learners’ 
understanding of essential concepts. Sometimes learners could attempt quizzes several times to 
gain mastery. Depending on the MOOC, learners received a new set of questions or answered the 
same questions again. Non-graded in-video quizzes were used for a similar objective and to 
increase engagement. Progressing to knowledge application, instructors used self-graded 
assessments, including self-reflections, and/or discussion questions. Carefully crafted rubrics, 
instructors emphasized, enabled learners to self-assess their submissions. 

Increasing in complexity and the degree of learner responsibility, peer-assessment was one of the 
most logistically challenging but cognitively promising types of assessment that all instructors 
intended to include in their MOOCs. The quality of peer feedback, learners' potential tendency to 
be lenient towards their peers, and language proficiency issues concerned some of the instructors. 
In practice, learners took the responsibility seriously and, even in the face of technical difficulties, 
provided feedback to their peers. Instructors from different disciplines believed that precise 
grading rubrics were important facilitators of effective peer-assessment. 
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MOOC communicative component: Strategizing instructional presence. 

Platform-based discussion forums were a common communicative component for all MOOCs that 
allowed learners to ask questions, engage in discussions, or seek help. Discussion forums also 
served as sources of information for the instructors to address technical or content-related 
problems identified by the learners. 

Instructors organized the discussion forum to ease posting comments or questions to relevant 
threads. Dedicating a thread to weekly discussion questions and video lectures proved effective 
although, as the instructors commented, some learners still posted their comments in irrelevant 
threads. 

Reasons for intervention and providing feedback included controversial discussions, strong 
disagreement, misconceptions, and comments posted to irrelevant discussions. 

Credit course vs. MOOC instruction. 

Instructors who answered a question about differences between the quality of learning in MOOCs 
versus credit courses agreed that finishing a U of T MOOC did not equate to completing a U of T 
credit course on the same topic. While U of T MOOCs were mostly informed by existing U of T 
credit courses, they were not as comprehensive due to the shorter length and perceived MOOC 
learners’ goals. For example, an instructor mentioned: “The question was how to take [a credit 
course] and break it into to, 15 minute bites that have a beginning, middle and end and covers 
something. It was a real re-envisioning of the course I had taught for so many years.” The 
difference in depth and breadth was also highlighted in assessment design. 

At least two instructors intended to encourage peer interaction as they practiced in their credit 
courses. Comparing expectations of MOOC learners and U of T students, one instructor 
mentioned: 


There was learning that was happening horizontally, student to 
student, rather than all the information coming top down. And 
that’s the way I approach my in-person classroom. A lot of it was 
about how to translate what I do in the classroom to this online 
environment? How to make the same kinds of things happen and 
address the challenges that will come from expanding it from 20 
participants in a seminar to 23,000 participants? 

A second factor that differentiated MOOC learning from credit course learning was the variety in 
form and the extent of feedback available to MOOC learners vs. U of T students. Students taking 
credit classes can ask questions and receive immediate feedback. Teaching assistants can also 
encourage students to consider alternative ways of approaching problems. Due to a far larger 
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ratio of learners to MOOC instructors and teaching assistants, providing feedback to every MOOC 
learner was impossible. Designing deliberate opportunities for soliciting learners’ understanding 
of a concept, and providing aggregate feedback that also acknowledged high quality learner 
contributions, was an effective strategy used by one of the instructors. 

Professional Implications of Teaching MOOCs 

Pedagogical implications. 

MOOC design and instruction entailed two main pedagogical implications: enhancing the clarity 
of content, e.g. discipline specific terminology; and restructuring instruction in credit courses. 
Instructors paid specific attention to making content or assessment criteria clear in their MOOCs. 
Regarding rubric development for peer-assessment, an instructor noted: 

In terms of the challenges with a MOOC, it is so hard to explain 
something extremely clearly and not have it be taken or 
interpreted in multiple ways... We had seven boxes where 
[learners] had to check... We wanted to make sure each of those 
points was really clear. 

Some of the instructors noted that they had devised new ways of articulating learning outcomes 
with clarity and precision, and likewise for the presentation of concepts in their credit courses. 
Such attention to clarity and shared understanding is captured in the following comment of one 
instructor: 


I have become much more clear about my learning objectives 
and articulating them and making sure that they are articulated 
in a language that students can understand - and I revisit them 
with the students. I think it makes the students more aware of 
what they are supposed to be learning. 

Restructuring credit course instruction resulted in instructors making MOOC lectures available to 
students and replacing a portion of lecture time with opportunities for active learning. Using 
MOOC material in credit courses happened either as pre-planned, where instructors intended to 
flip their classroom, or opportunistically. Some of the instructors did not integrate MOOC 
material into their teaching, as the material was not relevant to their credit courses. One 
affordance of flipped classrooms was described by an instructor as: “If you get it in the first five 
minutes, you still have to listen to all the questions and repetitions. Also in lectures there is a lot 
of pauses and examples that these students who got it in the first five minutes do not need.” 

Flipping a course appeared to be a fulfilling experience for instructors, as their students could 
come to class more prepared, spend more in-class time applying their knowledge to classroom 
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projects, or engage in peer instruction in small-group or whole-class format. Peer instruction, one 
instructor believed, reinforced students' learning as they had to articulate concepts in their own 
language. 

Recognition and dissemination. 

Instructors expressed their professional satisfaction with teaching MOOCs and appreciated the 
wide range of recognition they received. They highlighted three sources of recognition for their 
MOOC instruction: U of T top level administration including the Office of the President and the 
Office of the Vice President and Provost; the academic community, including U of T and faculty 
members from other universities; and MOOC learners. MOOC development also resulted in 
opportunities for broader dissemination of instructors’ scholarship and teaching. Some 
instructors had presented in international conferences and published papers based on their 
experience, thus gaining international academic exposure. 

Evolution of U of T MOOC Support Services 

Based on instructors’ accounts of available MOOC support and resources, OLS support targeted 
two aspects of MOOC development and implementation: pedagogical and technical. U of T MOOC 
support evolved and improved since 2012, which was evident from instructors’ descriptions of the 
nature and source of pedagogical and technological support. One instructor emphasized how the 
OLS team adjusted and improved their instructional support strategies since earlier U of T 
MOOCs. 

For the first cohort of U of T MOOC instructors in late 2012 and early 2013, support was mostly 
limited to technical aspects of MOOC design and development, and provided mainly by MOOC 
platform providers. Instructors added that OLS would follow up to make sure issues of concern 
were addressed. 

The second cohort of U of T MOOC instructors, mid-2013 to 2014, received both pedagogical and 
technological support from OLS. According to OLS documentation, planning and support 
provided research-informed instructional guidance for goal setting, content development, and 
assessment design through two-day course design institutes, workshops and consultations. The 
change in the level of institutional support for the second cohort of U of T MOOCs is evident in 
this instructor’s comment: 

OLS was fantastic in terms of the planning process... One of the 
things that OLS folks particularly pressed on me was the idea to 
have modules with themes, learning objectives for each module, 
and to also plan every little lecture bit. 

Another instructor described an example of technical support provided by OLS as: “[OLS learning 
strategist] helped a lot to sort out technical issues with [MOOC platform] and also he went in 
everyday and looked at participants' postings about technical problems.” These instructors 
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explained how following a “backward design”-inspired approach (encouraged through OLS) 
helped them systematically plan their MOOC and include as much detail in planning as to what 
each video lecture would contain in relation to weekly and overall learning outcomes. One of the 
instructors shared the story-board developed for each week. 

Measures of MOOC Success 

Completion rate, or the percentage of learners who obtain a certificate of completion, was deemed 
as a simplistic measure of MOOC success. The instructors believed learners’ goals would impact 
their level of engagement with learning, assessment, and communicative components of a MOOC. 
An instructor explained: 

There were many people in the course who did not want the 
certificate. Some learners would say that I am very happy. I am 
watching the videos and learning a lot. I am not doing the 
assessment because I don’t care about completion. I realized that 
there could be engaged people who don’t fall into that number 
for different reasons. 

Another instructor emphasized the relationship between learner motivation and learner activity 
when measuring the success of MOOCs. For one instructor, developing a high quality learning 
resource took precedence over completion rate. 

Quality, and not quantity, of posts in MOOC discussion forums was regarded as a more relevant 
measure of success. High quality contributions to forums partly showed that discussion questions 
and learning materials aroused learners’ interest and encouraged them to engage in knowledge- 
based discussions. 


Discussion and Conclusions 

We interviewed eight U of T instructors to understand their experience in MOOC design, 
development, and implementation, motivation to offer MOOCs, and potential impact on their 
credit course instruction. In this section and the section that follows, we discuss our findings in 
the light of existing literature, explain the contributions of this research, and suggest areas for 
further studies. 

Motivational Factors: Broader Outreach and Pedagogical Promise 

Findings of this study regarding instructors’ motivation to teach MOOCs echo previous research 
(Holland, & Tirthali, 2014). Multiple factors motivated U of T instructors to offer MOOCs. 
Identifying MOOCs as a means of reaching a broader international public and sharing a glimpse 
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of U of T teaching and learning approaches were main sources of motivation. Geographical 
distribution of U of T MOOC learners with respect to instructors' intended audience sometimes 
went beyond expectations for the instructors. Thus, establishing a broader outreach was one 
motivating goal that was realized. However, this motivation was contextualized within 
academically-driven motives such as examining ways to push the limits of pedagogy in large 
online classrooms, devising instructional approaches that leverage active learning within that 
context, and conceptualizing alternative learning opportunities. 

Instructors' emphasis on application, demonstrated in the learning outcomes section of our 
findings, resonates with this vision. Open courses can engage an academic audience beyond 
course boundaries by showcasing the latest research developments at U of T and cultivating 
research partnerships. The opportunity to develop and run a MOOC, for some instructors, 
coincided with their decision to flip their credit course. The possibility to connect a MOOC to 
current teaching and research interests justified the time and effort needed to design and 
implement MOOCs. 

MOOC Learning Outcomes: Moving beyond Knowledge Transmission 

Owing to the openness of MOOCs, characterizing MOOC learners in terms of their background, 
learning goals, and commitment level is near impossible (DeBoer, Ho, Stump, & Breslow, 2014). 
While U of T MOOC instructors echoed such concern, they did not compromise on MOOC 
learning outcomes. Mapping U of T MOOCs’ desired learning outcomes to Fink's (2003) 
Taxonomy of Significant Learning, instructors kept the bar high for MOOC learners by moving 
beyond a focus on foundational knowledge to fostering application and integration of knowledge. 
Higher level learning outcomes, such as "learning to learn" (Fink, 2003) were more relevant to U 
of T credit courses, due to the time it takes for learners to develop metacognitive knowledge. One 
instructor expressed his goal for on-campus students to have developed such competencies upon 
completing their undergraduate studies. 

Learning outcomes of a MOOC are articulated in relation to potential MOOC learners. For credit 
courses, an existing knowledge of students' characteristics facilitates the articulation of learning 
outcomes. MOOC instructors have to make assumptions about learners and plan ahead without 
extensive knowledge of the actual learners. This has raised concerns about MOOCs being teacher¬ 
centric (Eisenberg & Fischer, 2014). However, our findings provide evidence that MOOC 
instructors can engage learners in more than knowledge acquisition. Instructional design that 
demands active learning, even within the realities of MOOCs, can offset the inherent teacher- 
centered design. 

Working within Assessment Constraints 

U of T MOOC instructors used three types of formative and summative assessment relative to 
their learning outcomes: knowledge/understanding, self-assessment, and peer-assessment. The 
complexity of assessment design and implementation varied from machine-graded to peer-graded 
assessments. None of the instructors relied exclusively on information recall or machine-graded 
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quizzes. Rather, they aimed for educative assessment to measure students’ progress regarding 
application and integration of learning outcomes. 

To be effective, self and peer-assessment require rubrics that enable learners to evaluate their 
own or their peers' achievement of learning outcomes. The importance of rubric design is 
highlighted in the instructional design literature (Fink, 2003). Within a MOOC context, Kulkarni 
et al. (2013) emphasize the importance of creating readable rubrics and avoiding ambiguous 
terminology to facilitate peer feedback. U of T MOOC instructors highlighted the importance of 
rubric clarity to ensure shared understanding. As the sheer number of MOOC learners makes it 
impossible for the instructional team to check the accuracy of peer feedback, effective strategies to 
scaffold peer and self-assessment facilitate their scalability. Such scaffolding can be achieved 
through calibration exercises and corroboration with staff-graded assignments. In this study, one 
instructor mentioned employing calibration in a MOOC's peer-assessment assignment. 

Multi-Disciplinary MOOC Development Team 

Although the configuration of MOOC development teams varied, our findings were consistent 
with previous work on the importance of adopting a team approach to MOOC development 
(Belanger, & Thornton, 2013; Alario-Hoyos et al., 2014). Similar to Arnold et al. (2014), 
sometimes MOOC instructors opted for a "divide and conquer” approach to content development, 
although dealing with time constraints, rather than making the task manageable, was the reason 
for such choice. 

U of T MOOC instructors aimed to closely follow MOOC learners’ discussions to address issues of 
controversy or to provide feedback. The high volume of interactions during a MOOC offering 
becomes overwhelming for the instructional team (Robinson & Ash, 2014) and demands informal 
instructional support such as Community Teaching Assistants (CTAs). CTAs are learners who 
successfully finish a MOOC and demonstrate exceptional performance. If the same MOOC is re¬ 
offered, they are invited to provide peer feedback in threads within discussion forums. CTAs in a 
Coursera Python MOOC, for instance, addressed the bulk of learners' questions (Severance, 
2013). One U of T MOOC enlisted the help of CTAs in its second offering. Another instructor 
mentioned the anticipated involvement of CTAs in the second offering of a MOOC. Absence of 
CTAs presents a limitation for first time running MOOCs. Strategies such as inviting on-campus 
students to informally monitor the activities, as one U of T MOOC instructor did, may assist the 
instructional team in such circumstances. 

MOOC Development Support 

At this time, OLS provides central MOOC development support at U of T, while instructors 
depend on departmental facilities for technical support. The structure and centrality of support 
varies in different universities based on existing infrastructure prior to MOOC initiatives 
(Gooding, Klaas, Yager, & Kanchanaraksa, 2013). Here, we focus on the ways OLS has progressed 
at U of T and expanded its services to MOOC instructors since 2012. 
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Considering the variety of disciplines, anticipated learners, and learning outcomes of U of T 
MOOCs (Open UToronto MOOC Initiative: Report on Second Year of Activity, 2014), since 2012, 
OLS has gradually developed a common instructional design support system for all U of T MOOCs 
while also attending to specific requirements of each MOOC. In two OLS round-table sessions, 
instructors reviewed design considerations for online learning and engaged in hands-on 
instructional design tasks based on backward design and understanding by design frameworks 
(Wiggins & McTighe, 2005). These workshops highlighted the importance of the alignment 
between learning outcomes, assignments and activities, and assessment. Instructors received 
templates to visualize such alignments and had a chance to share a glimpse of their design with 
other workshop participants. The two instructors who participated in these workshops provided 
evidence of their effectiveness. 

Often times, faculty members undertake a MOOC on top of their teaching and research 
responsibilities. Pragmatic estimates of time and resource requirements, suggested by OLS, 
further facilitate their planning relative to their existing workload. Total time required for the 
instructional team to develop and teach a MOOC varies considerably in literature from an average 
of 75 hours (Gooding et al., 2013) to 620 hours (Belanger & Thornton, 2013). Contextual and 
institutional factors impact the time estimate, thus increasing the accuracy of the time 
commitment proposed based on existing U of T MOOC experience. 

Pedagogical Implications of MOOC Instruction 

U of T MOOC instructors deliberately provided opportunities for learners to engage with course 
material and with their peers through debates, discussion questions, and increasingly complex 
assignments. However, they did not equate the depth of MOOC learning with their credit courses 
for reasons such as shorter length of MOOCs, broader audience, and amount and quality of 
feedback available to MOOC learners vs. credit course students. 

MOOC development further stressed the importance of instructional design and upfront planning 
regarding the alignment of learning outcomes, learning activities, and assessment. Systematic 
upfront planning was emphasized more for the second cohort of U of T MOOC instructors as they 
participated in OLS MOOC development workshops. Comparing preparation for credit courses 
and MOOCs an instructor commented: 

I don’t think I ever spent as much time planning out any course. I 
obviously spend time thinking through how much time to devote 
to each section, but in terms of going down to every 5 minute 
chunk and what should I communicate by the end of that, I had 
never done that. 

Thus the MOOCs functioned much as the online for-credit courses in Power’s (2009) account, in 
bringing faculty that had taught for years to collaborate with instructional designers and reflect 
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on their pedagogical approach. Several of the MOOCs were also taught by teams of instructors, 
highlighting the importance of upfront planning and clarification of roles and responsibilities 
(Arnold et al., 2014). Bess (2000) suggested that the increased focus on hybrid and online courses 
in higher education would lead to an increase in course design teams with diverse backgrounds 
and specializations, which reflects MOOC design as we studied here. 

Characterizing MOOC Success 

Although MOOC attrition rates and the small proportion of learners who complete all assessment 
requirements has raised considerable concern (Breslow et al., 2013; Chen et ah, 2012), U of T 
instructors agreed that such raw numbers taken out of learner intent context do not reflect the 
success of a MOOC. More recently, learner engagement related to their intention and 
commitment level upon enrollment is used to propose an alternate interpretation of MOOC 
completion (DeBoer et ah, 2014; Kizilcec, Piece, & Schneider, 2013). Knowledge of MOOC learner 
intentions, though, is affected by learners' decisions to answer surveys designed to collect such 
information. 

Our participants noted the quality of learner participation and use of MOOC resources in light of 
learner intention as a relevant indicator of MOOC success. One instructor identified investigating 
strategies to keep learners engaged as a research opportunity in the MOOC context: 

In a MOOC it is so easy to stop going either because you are 
bored or because your life is too busy. It is still a glorious 
situation for looking at factors that enhance engagement. So if 
you can hold the students in that context where it is most easy to 
leave, then you are really doing something. Different techniques 
could be used and then compared. 


Implications and Future Research 


By focusing on less studied aspects of MOOCs as learning environments, MOOC design 
considerations and instructors' experience, our study addresses the paucity of research that could 
impact MOOC learner's experience. 

Three major findings from this research were: 

• Instructors' motivations to offer MOOCs were to provide quality learning experiences to 
broader groups of learners and to contribute to open educational resources. 

• MOOC experience led to increasing active learning opportunities in credit courses for 
four instructors. 
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• Systematic, research-based instructional support during MOOC design facilitated 
instructors' planning and their more nuanced understanding of instructional design 
processes. 

We propose three areas of future research: 

• design considerations and learning implications of self and peer-assessment for MOOCs; 

• impact of the organization of discussion forums and various arrangements of 
instructional team presence on learner participation in topic-related discussions; and 

• influence of MOOC planning and implementation on subsequent credit course design. 

We acknowledge the limitation of our study regarding confidentiality consideration that 
prevented us from comparing and contrasting instructional designs between MOOCs, or deeply 
contextualizing our findings with examples from specific MOOCs. 

U of T MOOC instructors were interested in designing new learning experiences for specific 
groups of learners inspired by the MOOC concept. Target learners varied from learners who could 
not enroll in a MOOC and complete it while the MOOC was running, known groups of 
professionals who were geographically dispersed in remote areas, and organizations seeking 
alternative professional development paths. 

On-demand MOOCs are already developed at U of T. Whether U of T MOOC instructors would 
design MOOCs with different learning tracks to correspond to learners’ intentions remains an 
open question. Currently, a study is underway at U of T that examines relations between learner 
intent, behavior, and outcome in multiple U of T MOOCs from various aspects of MOOC 
participation, such as forum activity. 
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